feat(evals): add eval suite for setup-shared-config-sync skill#331
Merged
Merged
Conversation
potiuk
force-pushed
the
eval-setup-shared-config-sync
branch
from
c3a8f80 to
70d9d9d
Compare
potiuk
approved these changes
May 27, 2026
potiuk
left a comment
potiuk
left a comment
Member
There was a problem hiding this comment.
LGTM — 11 cases across the two highest-signal steps in
setup-shared-config-sync (decide-action and draft-commit). Both
step-configs resolve to a real SKILL.md heading. Two prompt-injection
cases — one in the git-status output, one in a diff line — both
correctly preserve real-state output. Runner walks all 11 cases without
parse errors.
Smaller observations
- Both step-configs point to the same
## Walk-throughheading (the
only top-level heading covering the action flow), so the runner
extracts the same system-prompt content for both step evals. This
works — the model sees the full walk-through and the case data
differentiates which step is being exercised — but the precision is
lower than in the sibling eval suites (#332/#333/#334/#335), which
have distinct headings per step. If the skill's Walk-through ever
gets refactored into### Step N — …subheadings, the step-configs
should be retargeted accordingly. Not blocking.
This review was drafted by an AI-assisted tool and confirmed by an Apache Steward
maintainer. The maintainer approving this PR has read the findings and signed off.
If something feels off, please reply on the PR and a maintainer will follow up.More on how Apache Steward handles maintainer review:
CONTRIBUTING.md.
11 cases across 2 steps covering the action-path decision (in-sync, push-only, commit-then-push, pull-then-commit-then-push, not-a-git-repo, lock-held, injection resistance) and commit-message drafting (update existing script, add new config file, multi-file commit, injection in diff). step-3-decide-action cases are auto-comparable in --cli mode; step-5-draft-commit uses structural has_* flags for manual review. Updates tools/skill-evals/README.md suite count from 18 to 19. Generated-by: Claude (Opus 4.7)
potiuk
force-pushed
the
eval-setup-shared-config-sync
branch
from
70d9d9d to
b33ba47
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
11 cases across 2 steps covering the action-path decision (in-sync, push-only, commit-then-push, pull-then-commit-then-push, not-a-git-repo, lock-held, injection resistance) and commit-message drafting (update existing script, add new config file, multi-file commit, injection in diff). step-3-decide-action cases are auto-comparable in --cli mode; step-5-draft-commit uses structural has_* flags for manual review. Updates tools/skill-evals/README.md suite count from 18 to 19.
Generated-by: Claude (Opus 4.7)
Type of change
.claude/skills/<name>/) — eval fixtures updated belowtools/<system>/*.md)tools/*/withpyproject.toml)docs/,README.md,CONTRIBUTING.md)projects/_template/)prek, workflows, validators)Test plan
prek run --all-filespassesuv run pytest/ruff check/mypypasses(
PYTHONPATH=tools/skill-evals/src python3 -m skill_evals.runner tools/skill-evals/evals/<skill>/)(a regression test for the bug fixed / the behaviour added — see CONTRIBUTING.md)